Storage integration with an external storage location

ABSTRACT

A command to load or unload data at a storage location is received. In response to the command, a storage integration object associated with the storage location is identified. The storage integration object identifies a cloud identity object that corresponds to a cloud identity that is associated with a proxy identity object corresponding to a proxy identity granted permission to access the storage location. The data is loaded or unloaded at the storage location by assuming the proxy identity.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No.17/491,373 filed on Sep. 30, 2021, which is a Continuation of U.S.patent application Ser. No. 17/249,971 filed on Mar. 19, 2021 and issuedas U.S. Pat. No. 11,165,775 on Nov. 2, 2021, which is a Continuation ofU.S. patent application Ser. No. 16/911,850 filed on Jun. 25, 2020 andissued as U.S. Pat. No. 10,999,279 on May 4, 2021, which is aContinuation of U.S. patent application Ser. No. 16/683,641, filed Nov.14, 2019 and issued as U.S. Pat. No. 10,715,524 on Jul. 14, 2020, thecontents of which are hereby incorporated by reference in theirentireties.

TECHNICAL FIELD

Embodiments of the disclosure relate generally to a network-based datawarehouse and, more specifically, to external credential-less stages fordata warehouse storage integrations.

BACKGROUND

A cloud data warehouse (also referred to as a “network-based datawarehouse” or simply as a “data warehouse”) is a network-based systemused for data analysis and reporting that comprises a central repositoryof integrated data from one or more disparate sources. A cloud datawarehouse can store current and historical data that can be used forcreating analytical reports for an enterprise. To this end, datawarehouses typically provide business intelligence tools, tools toextract, transform, and load data into the repository, and tools tomanage and retrieve metadata.

External stages are components within a cloud data warehouse thatfacilitate integrations between a cloud data warehouse system and acustomer-managed storage location (referred to herein as “storageintegrations”). In general, external stages are used to load data to andunload data from customer-managed storage locations. In conventionalimplementations, external stages must be provided with secret securitycredentials to read data from and write data to these storage locations.However, the exchange of the secret security credentials createsvulnerabilities that may lead to exposure of the secret securitycredentials, which may lead to unauthorized access of data.Additionally, in conventional implementations, cloud data warehouseaccount administrators have limited ability to prohibit creation ofexternal stages by members in an organization, and an external stagecould potentially be used to exfiltrate confidential data to a personallocation. Further, storage owners do not have fine-grained control overaccess permissions for the storage locations. Conventional externalstages are also limited to use in a single file path and are not able tobe used in another file path, even if the credentials used to create theexternal stage are applicable to the other file path.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure will be understood more fully from the detaileddescription given below and from the accompanying drawings of variousembodiments of the disclosure.

FIG. 1 illustrates an example computing environment that includes anetwork-based data warehouse system in communication with a cloudstorage provider system, in accordance with some embodiments of thepresent disclosure.

FIG. 2 is a data flow diagram illustrating use of an externalcredential-less stage object within the computing environment to load orunload data at a storage location within the cloud storage providersystem to the network-based data warehouse system, in accordance withsome embodiments of the present disclosure.

FIG. 3 is an interaction diagram illustrating interactions betweencomponents in the computing environment in loading or unloading data ata storage location within the cloud storage provider system to thenetwork-based data warehouse system without exchanging securitycredentials associated with the storage location, in accordance withsome embodiments of the present disclosure.

FIGS. 4 and 5 are flow diagrams illustrating operations of thenetwork-based data warehouse in performing a method for loading orunloading data from an external storage platform using a credential-lessexternal stage object, in accordance with some embodiments of thepresent disclosure.

FIG. 6 is a block diagram illustrating components of a compute servicemanager, in accordance with some embodiments of the present disclosure.

FIG. 7 is a block diagram illustrating components of an executionplatform, in accordance with some embodiments of the present disclosure.

FIG. 8 illustrates a diagrammatic representation of a machine in theform of a computer system within which a set of instructions may beexecuted for causing the machine to perform any one or more of themethodologies discussed herein, in accordance with some embodiments ofthe present disclosure.

DETAILED DESCRIPTION

Reference will now be made in detail to specific example embodiments forcarrying out the inventive subject matter. Examples of these specificembodiments are illustrated in the accompanying drawings, and specificdetails are set forth in the following description in order to provide athorough understanding of the subject matter. It will be understood thatthese examples are not intended to limit the scope of the claims to theillustrated embodiments. On the contrary, they are intended to coversuch alternatives, modifications, and equivalents as may be includedwithin the scope of the disclosure.

As noted above, external stages are used in network-based datawarehouses to load data to and unload data from customer-managed storagelocations, and conventional external stages must be provided with secretsecurity credentials to enable access to these storage locations, whichcan create security vulnerabilities for the data. Aspects of the presentdisclosure address the above and other deficiencies of prior datawarehouse functionality by creating credential-less external stageobjects that do not require users to share secret security credentialswith a network-based data warehouse to facilitate loading and unloadingof data at storage locations in external cloud storage provider systems.The credential-less external state objects described herein also allowclient account administrators to prevent data exfiltration throughfine-grained control of access permissions.

Consistent with some embodiments, a network-based data warehouse createsan integration object comprising an identifier of a storage location(e.g., a universal resource locator (URL)) in a storage platform of anexternal cloud storage provider system (e.g., Amazon Web Services®(AWS), Microsoft Azure Blob Storage®, or Google Cloud Storage) to whichthe network-based data warehouse is to be provided access to load andunload data. The integration object further comprises an identifier of aproxy identity object maintained by the external cloud storage providersystem. Once created, the network-based data warehouse associates theintegration object with a cloud identity object that the cloud storageprovider system associates with the proxy identity object. The proxyidentity object defines a proxy identity that is granted access to thestorage location and may be assumed by the cloud identity object to loadand unload data at the storage location.

The data warehouse creates the integration object based on a command tocreate the storage integration. The command can be provided, forexample, by an administrative user of a client account the datawarehouse. The cloud identity object that is associated with theintegration object corresponds to the client account to which the userbelongs. A storage integration definition comprises the identifier ofthe storage location, the identifier of the proxy identity object, andan identifier of the cloud storage provider system. The storageintegration definition can, in some instances, further specify one ormore storage locations to which access is permitted or denied. Thestorage definition object can specify certain segments within thestorage location to which access is denied. For example, the storagelocation can be identified by a file path that corresponds to a storageresource within the storage platform such as a bucket or folder, and thecommand may specify a sub-folder within the file path to which access isdenied. In another example, the command may specify one or more filepaths to which access is permitted and in this example, access to allother file paths will be denied by default.

The data warehouse creates an external stage object based on the storageintegration object to load or unload data at the storage location. Theexternal stage object comprises the identifier of the storage locationand an identifier of the storage integration object. The data warehousecreates the external stage object based on a command to create theexternal stage object provided, for example, by the user that providedthe storage integration definition.

The network-based data warehouse can receive a command to load or unloaddata at the storage location. The command comprises an identifier of theexternal stage object. In response to the command, the data warehouseutilizes the external stage object to load or unload data at the storagelocation in the storage platform of the external cloud storage provider.In doing so, the network-based data warehouse uses security credentialsassociated with the cloud identity object to access credentials to allowthe cloud identity object to assume the proxy identity to load or unloadthe data. In this manner, the external stage object enables data to beloaded or unloaded at the storage location without exchanging securitycredentials associated with the storage location or storing the securitycredentials associated with the storage location with network-based datawarehouse system.

Credential-less external stage objects, as described herein, separatethe process of giving permissions to a storage location from the usageof that storage location to load and unload data. Credential-lessexternal stage objects also allow organizations to give permissions to anetwork-based data warehouse to use their data locations instead ofgiving secret credentials to the data warehouse. Organizations canspecify what roles may create and use storage locations for accessseparately from who may create and use stages set up in advance. Forinstance, an organization may allow account administrators to create aconnection to a storage location and because only the accountadministrators can create storage integrations, additional storageintegrations cannot be created to export data to thereby preventconfidential data exfiltration to unknown locations. Once created,non-administrative users can be granted permissions to read and writefrom fixed storage locations into an external stage object they create.A lower privilege user may only have the ability to use an existingstage.

Users with permissions to create a storage integration can control whatpaths under a base location can be accessed using that integration.Giving account administrators the ability to specify which users maycreate and use storage integrations allow an organization to controlwhere their internal data may flow to, or completely lock down dataexport altogether.

External credential-less stage objects also provide the benefit ofallowing access permissions to storage to be managed by the cloudstorage provider thereby allowing organizations utilizing the datawarehouse to leverage from their storage provider to manage data accessby the network-based data warehouse. If an account administrator decidesto revoke access by the data warehouse to a storage location, it can bedone immediately using the access controls provided by the storageprovider.

FIG. 1 illustrates an example computing environment 100 that includes anetwork-based data warehouse system 102 in communication with a cloudstorage provider system 104, in accordance with some embodiments of thepresent disclosure. To avoid obscuring the inventive subject matter withunnecessary detail, various functional components that are not germaneto conveying an understanding of the inventive subject matter have beenomitted from FIG. 1. However, a skilled artisan will readily recognizethat various additional functional components may be included as part ofthe computing environment 100 to facilitate additional functionalitythat is not specifically described herein.

As shown, the computing environment 100 comprises the network-based datawarehouse system 102 and a cloud storage provider system 104 (e.g.,AWS®, Microsoft Azure Blob Storage®, or Google Cloud Storage). Thenetwork-based data warehouse system 102 is a network-based system usedfor reporting and analysis of integrated data from one or more disparatesources including one or more storage locations within the cloud storageprovider system 104. The cloud storage provider system 104 comprises aplurality of computing machines and provides on-demand computer systemresources such as data storage and computing power to the network-baseddata warehouse system 102.

The network-based data warehouse system 102 comprises an accessmanagement system 110, a compute service manager 112, an executionplatform 114, and a database 116. The network-based data warehousesystem 102 hosts and provides data reporting and analysis services tomultiple client accounts. The access management system 110 enablesadministrative users of client accounts to manage access to resourcesand services provided by the network-based data warehouse system 102.Administrative users can create and manage identities (e.g., users,roles, and groups) and use permissions to allow or deny access to theidentities to resources and services.

The compute service manager 112 coordinates and manages operations ofthe network-based data warehouse system 102. The compute service manager112 also performs query optimization and compilation as well as managingclusters of computing services that provide compute resources (alsoreferred to as “virtual warehouses”). The compute service manager 112can support any number of client accounts such as end users providingdata storage and retrieval requests, system administrators managing thesystems and methods described herein, and other components/devices thatinteract with compute service manager 112.

The compute service manager 112 is also coupled to database 116, whichis associated with the data stored the computing environment 100. Thedatabase 116 stores data pertaining to various functions and aspectsassociated with the network-based data warehouse system 102 and itsusers. For example, the database 116 stores one or more credential-lessexternal stage objects 108. In general, an external stage object 108specifies a storage location (e.g., a URL) where data files are storedso that the data in the files can be loaded into a table storedinternally by the data warehouse 102 or so that data from a table can beunloaded into the data files stored internally by the data warehouse102. The one or more credential-less external stage objects 108 enablethe network-based data warehouse system 102 to access storage locationswithin the cloud storage provider system 104 without storing, using, orotherwise accessing security credentials associated with the storagelocations.

In some embodiments, the database 116 includes a summary of data storedin remote data storage systems as well as data available from a localcache. Additionally, the database 116 may include information regardinghow data is organized in remote data storage systems (e.g., the cloudstorage provider system 104) and the local caches. The database 116allows systems and services to determine whether a piece of data needsto be accessed without loading or accessing the actual data from astorage device.

The compute service manager 112 is further coupled to the executionplatform 114, which provides multiple computing resources that executevarious data storage and data retrieval tasks. The execution platform114 is coupled to storage platform 122 of the cloud storage providersystem 104. The storage platform 122 comprises multiple data storagedevices 124-1 to 124-N. In some embodiments, the data storage devices124-1 to 124-N are cloud-based storage devices located in one or moregeographic locations. For example, the data storage devices 124-1 to124-N may be part of a public cloud infrastructure or a private cloudinfrastructure. The data storage devices 124-1 to 124-N may be hard diskdrives (HDDs), solid state drives (SSDs), storage clusters, Amazon S3™storage systems or any other data storage technology. Additionally, thecloud storage provider system 104 may include distributed file systems(such as Hadoop Distributed File Systems (HDFS)), object storagesystems, and the like.

The execution platform 114 comprises a plurality of compute nodes. A setof processes on a compute node executes a query plan compiled by thecompute service manager 112. The set of processes can include: a firstprocess to execute the query plan; a second process to monitor anddelete micro-partition files using a least recently used (LRU) policyand implement an out of memory (OOM) error mitigation process; a thirdprocess that extracts health information from process logs and status tosend back to the compute service manager 112; a fourth process toestablish communication with the compute service manager 112 after asystem boot; and a fifth process to handle all communication with acompute cluster for a given job provided by the compute service manager112 and to communicate information back to the compute service manager112 and other compute nodes of the execution platform 114.

In addition to the storage platform 122, the cloud storage providersystem 104 also comprises an authentication and identity managementsystem 118. The authentication and identity management system 118 allowsusers to create and manage identities (e.g., users, roles, and groups)and use permissions to allow or deny access of the identities to cloudservices and resources. The access management system 110 of thenetwork-based data warehouse system 102 and the authentication andidentity management system 118 of the cloud storage provider system 104can communicate and share information so as to enable access andmanagement of resources and services shared by users of both thenetwork-based data warehouse system 102 and the cloud storage providersystem 104.

In some embodiments, communication links between elements of thecomputing environment 100 are implemented via one or more datacommunication networks. These data communication networks may utilizeany communication protocol and any type of communication medium. In someembodiments, the data communication networks are a combination of two ormore data communication networks (or sub-networks) coupled to oneanother. In alternate embodiments, these communication links areimplemented using any type of communication medium and any communicationprotocol.

As shown in FIG. 1, the data storage devices 124-1 to 124-N aredecoupled from the computing resources associated with the executionplatform 114. This architecture supports dynamic changes to thenetwork-based data warehouse system 102 based on the changing datastorage/retrieval needs as well as the changing needs of the users andsystems. The support of dynamic changes allows the network-based datawarehouse system 102 to scale quickly in response to changing demands onthe systems and components within the network-based data warehousesystem 102. The decoupling of the computing resources from the datastorage devices supports the storage of large amounts of data withoutrequiring a corresponding large amount of computing resources.Similarly, this decoupling of resources supports a significant increasein the computing resources utilized at a particular time withoutrequiring a corresponding increase in the available data storageresources.

The compute service manager 112, database 116, execution platform 114,storage platform 122, and authentication and identity management system118 are shown in FIG. 1 as individual discrete components. However, eachof the compute service manager 112, database 116, execution platform114, storage platform 122, and authentication and identity managementsystem 118 may be implemented as a distributed system (e.g., distributedacross multiple systems/platforms at multiple geographic locations).Additionally, each of the compute service manager 112, database 116,execution platform 114, storage platform 122, and authentication andidentity management system 118 can be scaled up or down (independentlyof one another) depending on changes to the requests received and thechanging needs of the network-based data warehouse system 102. Thus, inthe described embodiments, the network-based data warehouse system 102is dynamic and supports regular changes to meet the current dataprocessing needs.

During typical operation, the network-based data warehouse system 102processes multiple jobs determined by the compute service manager 112.These jobs are scheduled and managed by the compute service manager 112to determine when and how to execute the job. For example, the computeservice manager 112 may divide the job into multiple discrete tasks andmay determine what data is needed to execute each of the multiplediscrete tasks. The compute service manager 112 may assign each of themultiple discrete tasks to one or more nodes of the execution platform114 to process the task. The compute service manager 112 may determinewhat data is needed to process a task and further determine which nodeswithin the execution platform 114 are best suited to process the task.Some nodes may have already cached the data needed to process the taskand, therefore, be a good candidate for processing the task. Metadatastored in the database 116 assists the compute service manager 112 indetermining which nodes in the execution platform 114 have alreadycached at least a portion of the data needed to process the task. One ormore nodes in the execution platform 114 process the task using datacached by the nodes and, if necessary, data retrieved from the cloudstorage provider system 104. It is desirable to retrieve as much data aspossible from caches within the execution platform 114 because theretrieval speed is typically much faster than retrieving data from thecloud storage provider system 104.

As shown in FIG. 1, the computing environment 100 separates theexecution platform 114 from the storage platform 122. In thisarrangement, the processing resources and cache resources in theexecution platform 114 operate independently of the data storage devices124-1 to 124-n in the cloud storage provider system 104. Thus, thecomputing resources and cache resources are not restricted to specificdata storage devices 124-1 to 124-n. Instead, all computing resourcesand all cache resources may retrieve data from, and store data to, anyof the data storage resources in the cloud storage provider system 104.

FIG. 2 is a data flow diagram illustrating use of an external stageobject 200 within the computing environment 100, in accordance with someembodiments of the present disclosure. The external stage object 200 isan example of the credential-less external stage object(s) 108illustrated in FIG. 1. The external stage object 200 is generated by thecompute service manager 112 and stored in the database 116. The externalstage object 200 is generated by the compute service manager 112 withina client account 204. The compute service manager 112 creates theexternal stage object 200 based on input received from a computingdevice in communication with the network-based data warehouse system102. For example, a user 206 of the client account 204 can utilize acommand line or other user interface provided to a computing device 206by the network-based data warehouse system 102 to provide a command tocreate the external stage object 200.

The external stage object 200 is a component used to load or unload dataat a storage location within the storage platform 122 to thenetwork-based data warehouse system 102. In this particular example, theexternal stage object 200 specifies a storage location corresponding toa storage resource 208 within the storage platform 122 as a locationfrom which data can be loaded or unloaded. The storage resource 208resides on one or more of the storage devices 124-1 to 124-N of thestorage platform 122. The external stage object 200 further comprises areference (e.g., a pointer) to a storage integration object 202.

The storage integration object 202 is created within the client account204 by the compute service manager 112 and is stored within the database116. The compute service manager 112 creates the external stage object200 based on input received from the computing device 206 of the user205 of the client account 204 in communication with the network-baseddata warehouse system 102. For example, the user 206 can utilize thecommand line or other user interface provided to the computing device206 by the network-based data warehouse system 102 to provide a commandto create the storage integration object 202.

It shall be appreciated that the user who provides the command to createthe external stage object 200 may be a different user from the user thatprovides the command to create the storage integration object 202. Forexample, a first user with administrator privileges—an administrativeuser—may provide the command to create the storage integration object202 and as part of the command, may grant permission to a second user touse the storage integration object 202 to create external stage objects.In this example, the second user may provide the command to create theexternal stage object 200.

The storage integration object 202 defines a storage integration betweenthe network-based data warehouse system 102 and an externally managedstorage location in the storage platform 122. More specifically, thestorage integration object 202 describes properties of a storageintegration between the network-based data warehouse system 102 and thecustomer managed storage resource 208 (e.g., a folder, data bucket, orother storage resource). The storage integration object 202 comprises anidentifier of a storage location corresponding to the storage resource208 (e.g., a URL) and an identifier of the cloud storage provider system104. In some embodiments, the storage integration object 202 may furtherspecify one or more storage locations to which access to data is to bedenied. For example, the external stage object 200 may identify a basestorage location to which access is to be allowed using a file path andthe storage integration object 202 may further identify a portion of thebase storage location to which access is to be allowed or denied with asub-path of the file path.

Once created, the compute service manager 112 associates the storageintegration object 202 with a cloud identity object 210 within a serviceaccount 212 maintained by the authentication and identity managementsystem 118 that is associated with the network-based data warehousesystem 102 and the client account 204. The cloud identity object 210 isan identity within the cloud storage provider system 104 associated withthe client account 204. The cloud identity object 210 may be createdwhen the client account 204 is created. A unique identifier (e.g., anAmazon Resource Name (ARN)) is associated with the cloud identity object210 at creation. A storage provider administrator can utilize theauthentication and identity management system 118 to grant permission tothe cloud identity object 210 to access storage using the identifier ofthe cloud identity object 210.

The compute service manager 112 may store cloud storage provideridentity identifiers in the database 116 in an encrypted format. Thecompute service manager 112 may further store security credentialsassociated with each cloud storage provider identity in the database 116in an encrypted format.

The cloud storage provider system 104 generates a proxy identity object214 within a client account 216 of the cloud storage provider system104. The client account 216 is the account of the client correspondingto the client account 204 within the cloud storage provider system 104.The cloud storage provider system 104 generates the proxy identityobject 214 based on input specified by an administrative user of theclient account 216. In some instances, the administrative user of theclient account 216 is the user 205.

The proxy identity object 214 defines a proxy identity with anassociated trust policy for making services requests within the cloudstorage provider system 104. More specifically, the proxy identityobject 214 includes a set of permissions that allow the cloud identityobject 210 to assume the proxy identity to read data from and write datato the storage resource 208. Rather than being uniquely associated witha single person like a user, the proxy identity object 214 defines aproxy identity that can be assumed by multiple users.

In some instances, the proxy identity defined by the proxy identityobject 214 does not have long-term security credentials, and in theseinstances, another identity that is assuming the proxy identity utilizestemporary security credentials provided by the authentication andidentity management system 118 to access the proxy identity. Consistentwith these embodiments, the temporary security credentials may expireafter an expiration time.

The cloud storage provider system 104 assigns a unique identifier to theproxy identity object 214 (e.g., an Amazon® Resource Name (ARN)). Theunique identifier of the proxy identity object 214 is used by thestorage administrator to grant access to storage.

In response to receiving a command to load data from the storagelocation corresponding to the storage resource 208 to an internallymanaged storage resource (e.g., a table) or to unload data from theinternally managed storage resource to the storage locationcorresponding to the storage resource 208, the network-based datawarehouse system 102 uses the external stage object 200 to load orunload the data. In particular, the compute service manager 112identifies and accesses the storage integration object 202 using theexternal stage object 200 and uses the storage integration object 202 toaccess security credentials associated with the cloud identity object210. The compute service manager 112 uses security credentialsassociated with the cloud identity object 210 to access securitycredentials from the authentication and identity management system 118to allow the cloud identity object 210 to assume the proxy identitydefined by the proxy identity object 214 to load or unload data betweenthe internal storage resource and the storage resource 208.

FIG. 3 is an interaction diagram illustrating interactions between thenetwork-based data warehouse system 102 and the cloud storage providersystem 104 in a method 300 for using the external stage object 200 toload or unload data at the storage resource 208 within the cloud storageprovider system 104 to the network-based data warehouse system 102, inaccordance with some embodiments of the present disclosure. For ease ofexplanation, the method 300 is described below with reference to thecomponents illustrated in FIGS. 1 and 2 and described above.

At operation 302, the cloud storage provider system 104 creates theproxy identity object 214 within the client account 216. The cloudstorage provider system 104 generates the proxy identity object 214based on input specified by an administrative user of the client account216. As noted above, the proxy identity object 214 includes a set ofpermissions that allow the cloud identity object 210 to assume the proxyidentity to read data from and write data to the storage resource 208.

At operation 304, the compute service manager 112 of the network-baseddata warehouse system 102 creates the storage integration object 202 inthe database 116. The compute service manager 112 creates the storageintegration object 202 based on a command provided by a first user(e.g., the user 206) of the client account 204. The first user may be anadministrative user of the client account 204. As noted above, thestorage integration object 202 comprises: an integration name, anidentifier of an externally managed storage location such as a URLcorresponding to the storage resource 208; an identifier of the cloudstorage provider system 104; and an identifier of the proxy identityobject 214. In some embodiments, the storage integration object 200 mayfurther specify one or more storage locations to which access to data isto be denied. The one or more storage locations to which access to datais to be denied may correspond to portions of the storage locations towhich access to the network-based data warehouse system 102 is to beprovided. For example, the external stage object 200 may identity a basestorage location to which access is to be allowed using a file path andthe storage integration object 200 may further identify a portion of thebase storage location to which access is to be denied with a sub-path ofthe file path.

At operation 306, the compute service manager 112 associates the storageintegration object 202 with the cloud identity object 210. The computeservice manager 112 associates the storage integration object 202 withthe cloud identity object 210 based on an association between the clientaccount 204 and the cloud identity object 210.

At operation 308, the access management system 110 sets usagepermissions for the storage integration object 202. The setting of usagepermissions for the integration object may comprise granting a seconduser permission to use the storage integration object. In someembodiments, the access management system 110 may grant usage permissionto an identity that corresponds to multiple users thereby providingusage permission to each user associated with the identity.

At operation 310, the compute service manager 112 creates the externalstage object 200. The compute service manager 112 creates the externalstage object 200 based on an external stage creation command to load orunload data at a storage location. The storage location may be the sameas the storage location specified in the storage integration object 202or may comprise a portion of the storage location specified in thestorage integration object 202. The external stage object 200 comprisesan identifier corresponding to the storage location (e.g., the URLcorresponding to the storage location) and a reference to the storageintegration object 202 (e.g., a pointer).

At operation 312, the access management system 110 sets usagepermissions for the external stage object 200. The access managementsystem 110 may set usage permissions based on input received from thesecond user. The setting of usage permissions for the integration objectmay comprise granting usage permission to a third identity. The thirdidentity may correspond to a single user or may be associated withmultiple users. Once usage permission is granted to the third identity,the one or more users associated with the third identity are allowed touse the external stage object to load or unload data.

At operation 314, a cloud storage provider administrator sets usagepermission on the cloud storage provider system 104 for the proxyidentity object. In setting the usage permissions for the proxy identityobject, the cloud storage provider system 104 grants permission to thecloud identity object to use the proxy identity object to load andunload data at the storage location.

At operation 316, the compute service manager 112 receives a command toload data from the storage location to an internally managed storageresource (e.g., a table) or to unload data from the internally managedstorage resource to the storage location. The command comprises theidentifier of the external stage object 200 (e.g., the integrationname). The command may be received from a computing device of a thirduser associated with the third identity. In response to the command, thecompute service manager 112 uses the external stage object 200 toexecute the command, at operation 318. In executing the command, thenetwork-based data warehouse system 102 authenticates with the proxyidentity object 214 using security credentials associated with the proxyidentity object 214 to assume the proxy identity defined by the proxyidentity object 214. The network-based data warehouse system 102 assumesthe proxy identity object 214 to either load data from the storagelocation to an internally managed storage resource (e.g., a table) or tounload data from the internally managed storage resource to the storagelocation, at operation 322.

FIGS. 4 and 5 are flow diagrams illustrating operations of thenetwork-based data warehouse system 102 in performing a method 400 forloading or unloading data from the storage platform 122 using theexternal stage object 200, in accordance with some embodiments of thepresent disclosure. The method 400 may be embodied in computer-readableinstructions for execution by one or more hardware components (e.g., oneor more processors) such that the operations of the method 400 may beperformed by components of network-based data warehouse system 102.Accordingly, the method 400 is described below, by way of example withreference thereto. However, it shall be appreciated that the method 400may be deployed on various other hardware configurations and is notintended to be limited to deployment within the network-based datawarehouse system 102.

At operation 405, the compute service manager 112 receives a command tocreate a storage integration object (also referred to as a “storageintegration creation command”). The storage integration creation commandis received from a computing device in communication with the datawarehouse 102 and is specified by a first user via a command line or UIprovided to the computing device by the network-based data warehousesystem 102. The first user is an administrative user belonging to aclient account of the network-based data warehouse system 102. Ingeneral, the command specifies storage locations that are allowed orblocked as part of a storage integration. For example, the command cancomprise an identifier corresponding to a first storage location (e.g.,a URL) in the storage platform 122 of the cloud storage provider system104 to which the network-based data warehouse is to be allowed access toload and unload data. The first storage location corresponds to a datastorage resource such as a data folder or a data bucket. The storageintegration creation command further specifies the cloud storageprovider system 104, an integration name, and an identifier of a proxyidentity object maintained by the cloud storage provider system 104. Theproxy identity object defines a proxy identity that has associatedpermissions to access the first storage location. The proxy identityobject further includes permissions that allow a cloud identity toassume the proxy identity to load and unload data at the first storagelocation.

In some instances, the first storage location may be a base storagelocation and the storage integration creation command can furtherspecify one or more blocked storage locations to which access by thenetwork-based data warehouse system 102 is denied. The blocked locationsmay correspond to one or more portions of the first storage location.For example, the first storage location may correspond to a storagebucket (e.g., an S3 storage bucket) and the storage integration creationcommand may specify one or more folders within the storage bucket towhich access is to be denied.

At operation 410, the compute service manager 112 creates a storageintegration object in the database 116 based on the storage integrationcreation command. The storage integration object specifies the firststorage location (e.g., a URL), the cloud storage provider system 104,and the proxy identity object. More specifically, the storageintegration object comprises a first identifier corresponding to thefirst storage location (e.g., a URL), an identifier of the cloud storageprovider system 104, and an identifier of the proxy identity object(e.g., an ARN). Once created, the network-based data warehouse system102, at operation 415, associates the integration object with a cloudidentity object that is associated with the proxy identity object. Thecloud identity object defines a cloud identity used to access cloudservices provided by the cloud storage provider system 104. The cloudidentity object is associated with the storage integration object basedon an association between the cloud identity object and the clientaccount to which the first user belongs.

At operation 420, the access management system 110 sets usagepermissions for the storage integration object. The access managementsystem 110 may set usage permissions based on input received from thefirst user. The setting of usage permissions for the integration objectmay comprise granting a second identity permission to use the storageintegration object to create one or more external stage objects. In someembodiments, the second identity corresponds to at least a second user.In some embodiments, the access management system 110 may grant usagepermission to an identity that corresponds to multiple users therebyproviding usage permission to each user associated with the identity.

At operation 425, the compute service manager 112 receives a command tocreate an external stage object (also referred to as a “external stagecreation command”). The external stage creation command is received froma computing device in communication with the data warehouse 102 and isspecified by the second user via a command line or UI provided to thecomputing device by the network-based data warehouse system 102. Theexternal stage creation command comprises an identifier of a secondstorage location (e.g., a URL) and an identifier of the storageintegration object (e.g., the name of the storage integration object).In some instances, the second storage location may be the same as thefirst storage location while in other instances, the second storagelocation may correspond to a location within the first storage location.That is, the second storage location may correspond to a portion of thefirst storage location.

At operation 430, the compute service manager 112 creates an externalstage object based on the external stage creation command to load orunload data at the second storage location. The external stage objectidentifies the second storage location and includes an association withthe storage integration object. More specifically, the external stageobject comprises the identifier of the second storage location (e.g.,the URL corresponding to the second storage location) and a reference tothe storage integration object (e.g., a pointer).

At operation 435, the access management system 110 sets usagepermissions for the external stage object. The access management system110 may set usage permissions based on input provided by the seconduser. The setting of usage permissions for the integration object maycomprise granting usage permission to a third identity associated withat least a third user. Once usage permission is granted to the thirdidentity, at least the third user is allowed to use the external stageobject.

At operation 440, the compute service manager 112 receives a command tounload from an internal data resource (e.g., a table) to a third storagelocation or to load data from the third storage location to the internaldata resource. The command comprises the identifier of the externalstage object (e.g., the name given to the external stage object). Thecommand may be received from a computing device of a third user. Inresponse to the command, the compute service manager 112, at operation445, loads or unloads the data at the third storage location in thestorage platform 122 of the external cloud storage provider using theexternal stage object 200. In a first example, the compute servicemanager 112 copies data from the storage location to a table maintainedin a storage location that is internal to the network-based datawarehouse system 102. In a second example, the compute service manager112 copies data from a table maintained in an internal storage locationto the storage location specified in the command. The third storagelocation corresponds to the second storage location. For example, thethird storage location may be the same as the second storage location ormay correspond to a location within the second storage location.

As shown in FIG. 5, the method 400 may, in some embodiments, furtherinclude operations 505, 510, 515, 520, and 525. Consistent with theseembodiments, the operations 505 and 510 may be performed subsequent tothe operation 440 where compute service manager 112 receives the commandto load or unload data at the storage location.

At operation 505, the compute service manager 112 identifies the storageintegration object based on an association with the external stageobject. For example, as noted above, the external stage objectreferenced in the command to load or unload data includes a reference tothe integration object (e.g., a pointer).

At operation 510, the compute service manager 112 works in conjunctionwith the access management system 110 to verify user permissionsassociated with the third user. In verifying the user permissionsassociated with the third user, the compute service manager 112 verifiesthe third user has permission to use the external stage object and thestorage integration object.

Consistent with these embodiments, the operations 515, 520, and 525 maybe performed as part of operation 445 (e.g., as a sub-routine orsub-operations) where the compute service manager 112 loads or unloadsdata at the storage location.

At operation 515, the compute service manager 112 verifies that thethird storage location is allowed by the storage integration object.That is, the compute service manager 112 checks the third storagelocation against the first storage location identified in the storageintegration object to verify that the third storage location is withinthe first storage location. The service manager 112 may further checkthe third storage location against any blocked storage locationsspecified by the storage integration object to determine whether thethird storage location is allowed by the storage integration object.

At operation 520, the compute service manager 112 accesses securitycredentials to be used in authenticating with the cloud storage providersystem 104 to assume the proxy identity defined by the proxy identityobject. In some embodiments, the security credentials are temporary andmay expire after a time limit is reached (e.g., 1 hour) and may belimited in scope for use specifically in loading or unloading data atthe first storage location.

Consistent with some embodiments, the compute service manager 112 mayobtain the security credentials by transmitting a request to theauthentication and identity management system 118 of the cloud storageprovider system 104 for the security credentials. The request cancomprise or indicate a first identifier corresponding to the cloudidentity object, a second identifier corresponding to proxy identityobject, and security credentials associated with the cloud identityobject. The security credentials associated with the cloud identityobject may be stored in an encrypted format in the database 116. Theauthentication and identity management system 118 of the cloud storageprovider system 104 provides the security credentials in response to therequest.

At operation 525, the compute service manager 112 causes the cloudidentity to assume the proxy identity defined by the proxy identityobject using the credentials obtained from the authentication andidentity management system 118. That is, the cloud identity interactswith the storage location (e.g., by loading or unloading data) via theproxy identity and may exchange data with the compute service manager112 using the proxy identity.

FIG. 6 is a block diagram illustrating components of the compute servicemanager 112, in accordance with some embodiments of the presentdisclosure. As shown in FIG. 6, request processing service 602 managesreceived data storage requests and data retrieval requests (e.g., jobsto be performed on database data). For example, the request processingservice 602 may determine the data necessary to process a received query(e.g., a data storage request or data retrieval request). The data maybe stored in a cache within the execution platform 114 or in a datastorage device in cloud storage provider system 104.

A management console service 604 supports access to various systems andprocesses by administrators and other system managers. Additionally, themanagement console service 604 may receive a request to execute a joband monitor the workload on the system.

The compute service manager 112 also includes a job compiler 606, a joboptimizer 608 and a job executor 610. The job compiler 606 parses a jobinto multiple discrete tasks and generates the execution code for eachof the multiple discrete tasks. The job optimizer 608 determines thebest method to execute the multiple discrete tasks based on the datathat needs to be processed. The job optimizer 608 also handles variousdata pruning operations and other data optimization techniques toimprove the speed and efficiency of executing the job. The job executor610 executes the execution code for jobs received from a queue ordetermined by the compute service manager 112.

A job scheduler and coordinator 612 sends received jobs to theappropriate services or systems for compilation, optimization, anddispatch to the execution platform 114. For example, jobs may beprioritized and processed in that prioritized order. In an embodiment,the job scheduler and coordinator 612 determines a priority for internaljobs that are scheduled by the compute service manager 112 with other“outside” jobs such as user queries that may be scheduled by othersystems in the database but may utilize the same processing resources inthe execution platform 114. In some embodiments, the job scheduler andcoordinator 612 identifies or assigns particular nodes in the executionplatform 114 to process particular tasks. A virtual warehouse manager614 manages the operation of multiple virtual warehouses implemented inthe execution platform 114. As discussed below, each virtual warehouseincludes multiple execution nodes that each include a cache and aprocessor.

Additionally, the compute service manager 112 includes a configurationand metadata manager 616, which manages the information related to thedata stored in the remote data storage devices and in the local caches(e.g., the caches in execution platform 114). The configuration andmetadata manager 616 uses the metadata to determine which datamicro-partitions need to be accessed to retrieve data for processing aparticular task or job. A monitor and workload analyzer 618 overseeprocesses performed by the compute service manager 112 and manages thedistribution of tasks (e.g., workload) across the virtual warehouses andexecution nodes in the execution platform 114. The monitor and workloadanalyzer 618 also redistribute tasks, as needed, based on changingworkloads throughout the data warehouse 102 and may further redistributetasks based on a user (e.g., “external”) query workload that may also beprocessed by the execution platform 114. The configuration and metadatamanager 616 and the monitor and workload analyzer 618 are coupled to adata storage device 620. Data storage device 620 in FIG. 6 represent anydata storage device within the data warehouse 102. For example, datastorage device 620 may represent caches in execution platform 114,storage devices in cloud storage provider system 104, or any otherstorage device.

FIG. 7 is a block diagram illustrating components of the executionplatform 114, in accordance with some embodiments of the presentdisclosure. As shown in FIG. 7, the execution platform 114 includesmultiple virtual warehouses, including virtual warehouse 1, virtualwarehouse 2, and virtual warehouse n. Each virtual warehouse includesmultiple execution nodes that each include a data cache and a processor.The virtual warehouses can execute multiple tasks in parallel by usingthe multiple execution nodes. As discussed herein, the executionplatform 114 can add new virtual warehouses and drop existing virtualwarehouses in real-time based on the current processing needs of thesystems and users. This flexibility allows the execution platform 114 toquickly deploy large amounts of computing resources when needed withoutbeing forced to continue paying for those computing resources when theyare no longer needed. All virtual warehouses can access data from anydata storage device (e.g., any storage device in cloud storage providersystem 104).

Although each virtual warehouse shown in FIG. 7 includes three executionnodes, a particular virtual warehouse may include any number ofexecution nodes. Further, the number of execution nodes in a virtualwarehouse is dynamic, such that new execution nodes are created whenadditional demand is present, and existing execution nodes are deletedwhen they are no longer necessary.

Each virtual warehouse is capable of accessing any of the data storagedevices 124-1 to 124-n shown in FIG. 1. Thus, the virtual warehouses arenot necessarily assigned to a specific data storage device 124-1 to124-n and, instead, can access data from any of the data storage devices124-1 to 124-n within the cloud storage provider system 104. Similarly,each of the execution nodes shown in FIG. 7 can access data from any ofthe data storage devices 124-1 to 124-n. In some embodiments, aparticular virtual warehouse or a particular execution node may betemporarily assigned to a specific data storage device, but the virtualwarehouse or execution node may later access data from any other datastorage device.

In the example of FIG. 7, virtual warehouse 1 includes three executionnodes 702-1, 702-2, and 702-n. Execution node 702-1 includes a cache704-1 and a processor 706-1. Execution node 702-2 includes a cache 704-2and a processor 706-2. Execution node 702-n includes a cache 704-n and aprocessor 706-n. Each execution node 702-1, 702-2, and 702-n isassociated with processing one or more data storage and/or dataretrieval tasks. For example, a virtual warehouse may handle datastorage and data retrieval tasks associated with an internal service,such as a clustering service, a materialized view refresh service, afile compaction service, a storage procedure service, or a file upgradeservice. In other implementations, a particular virtual warehouse mayhandle data storage and data retrieval tasks associated with aparticular data storage system or a particular category of data.

Similar to virtual warehouse 1 discussed above, virtual warehouse 2includes three execution nodes 712-1, 712-2, and 712-n. Execution node712-1 includes a cache 714-1 and a processor 716-1. Execution node 712-2includes a cache 714-2 and a processor 716-2. Execution node 712-nincludes a cache 714-n and a processor 716-n. Additionally, virtualwarehouse 3 includes three execution nodes 722-1, 722-2, and 722-n.Execution node 722-1 includes a cache 724-1 and a processor 726-1.Execution node 722-2 includes a cache 724-2 and a processor 726-2.Execution node 722-n includes a cache 724-n and a processor 726-n.

In some embodiments, the execution nodes shown in FIG. 7 are statelesswith respect to the data the execution nodes are caching. For example,these execution nodes do not store or otherwise maintain stateinformation about the execution node or the data being cached by aparticular execution node. Thus, in the event of an execution nodefailure, the failed node can be transparently replaced by another node.Since there is no state information associated with the failed executionnode, the new (replacement) execution node can easily replace the failednode without concern for recreating a particular state.

Although the execution nodes shown in FIG. 7 each includes one datacache and one processor, alternate embodiments may include executionnodes containing any number of processors and any number of caches.Additionally, the caches may vary in size among the different executionnodes. The caches shown in FIG. 7 store, in the local execution node,data that was retrieved from one or more data storage devices in cloudstorage provider system 104. Thus, the caches reduce or eliminate thebottleneck problems occurring in platforms that consistently retrievedata from remote storage systems. Instead of repeatedly accessing datafrom the remote storage devices, the systems and methods describedherein access data from the caches in the execution nodes which issignificantly faster and avoids the bottleneck problem discussed above.In some embodiments, the caches are implemented using high-speed memorydevices that provide fast access to the cached data. Each cache canstore data from any of the storage devices in the cloud storage providersystem 104.

Further, the cache resources and computing resources may vary betweendifferent execution nodes. For example, one execution node may containsignificant computing resources and minimal cache resources, making theexecution node useful for tasks that require significant computingresources. Another execution node may contain significant cacheresources and minimal computing resources, making this execution nodeuseful for tasks that require caching of large amounts of data. Yetanother execution node may contain cache resources providing fasterinput-output operations, useful for tasks that require fast scanning oflarge amounts of data. In some embodiments, the cache resources andcomputing resources associated with a particular execution node aredetermined when the execution node is created, based on the expectedtasks to be performed by the execution node.

Additionally, the cache resources and computing resources associatedwith a particular execution node may change over time based on changingtasks performed by the execution node. For example, an execution nodemay be assigned more processing resources if the tasks performed by theexecution node become more processor-intensive. Similarly, an executionnode may be assigned more cache resources if the tasks performed by theexecution node require a larger cache capacity.

Although virtual warehouses 1, 2, and n are associated with the sameexecution platform 114, the virtual warehouses may be implemented usingmultiple computing systems at multiple geographic locations. Forexample, virtual warehouse 1 can be implemented by a computing system ata first geographic location, while virtual warehouses 2 and n areimplemented by another computing system at a second geographic location.In some embodiments, these different computing systems are cloud-basedcomputing systems maintained by one or more different entities.

Additionally, each virtual warehouse is shown in FIG. 7 as havingmultiple execution nodes. The multiple execution nodes associated witheach virtual warehouse may be implemented using multiple computingsystems at multiple geographic locations. For example, an instance ofvirtual warehouse 1 implements execution nodes 702-1 and 702-2 on onecomputing platform at a geographic location and implements executionnode 702-n at a different computing platform at another geographiclocation. Selecting particular computing systems to implement anexecution node may depend on various factors, such as the level ofresources needed for a particular execution node (e.g., processingresource requirements and cache requirements), the resources availableat particular computing systems, communication capabilities of networkswithin a geographic location or between geographic locations, and whichcomputing systems are already implementing other execution nodes in thevirtual warehouse.

Execution platform 114 is also fault tolerant. For example, if onevirtual warehouse fails, that virtual warehouse is quickly replaced witha different virtual warehouse at a different geographic location.

A particular execution platform 114 may include any number of virtualwarehouses. Additionally, the number of virtual warehouses in aparticular execution platform is dynamic, such that new virtualwarehouses are created when additional processing and/or cachingresources are needed. Similarly, existing virtual warehouses may bedeleted when the resources associated with the virtual warehouse are nolonger necessary.

In some embodiments, the virtual warehouses may operate on the same datain cloud storage provider system 104, but each virtual warehouse has itsown execution nodes with independent processing and caching resources.This configuration allows requests on different virtual warehouses to beprocessed independently and with no interference between the requests.This independent processing, combined with the ability to dynamicallyadd and remove virtual warehouses, supports the addition of newprocessing capacity for new users without impacting the performanceobserved by the existing users.

FIG. 8 illustrates a diagrammatic representation of a machine 800 in theform of a computer system within which a set of instructions may beexecuted for causing the machine 800 to perform any one or more of themethodologies discussed herein, according to an example embodiment.Specifically, FIG. 8 shows a diagrammatic representation of the machine800 in the example form of a computer system, within which instructions816 (e.g., software, a program, an application, an applet, an app, orother executable code) for causing the machine 800 to perform any one ormore of the methodologies discussed herein may be executed. For example,the instructions 816 may cause the machine 800 to execute any one ormore operations of any one or more of the method 300 or 400. As anotherexample, the instructions 816 may cause the machine 800 to implementedportions of the data flows illustrated in any one or more of FIGS. 3-5.In this way, the instructions 816 transform a general, non-programmedmachine into a particular machine 800 (e.g., the access managementsystem 110, the compute service manager 112, the execution platform 114,the authentication and identity management system 118, and the computingdevices 206) that is specially configured to carry out any one of thedescribed and illustrated functions in the manner described herein.

In alternative embodiments, the machine 800 operates as a standalonedevice or may be coupled (e.g., networked) to other machines. In anetworked deployment, the machine 800 may operate in the capacity of aserver machine or a client machine in a server-client networkenvironment, or as a peer machine in a peer-to-peer (or distributed)network environment. The machine 800 may comprise, but not be limitedto, a server computer, a client computer, a personal computer (PC), atablet computer, a laptop computer, a netbook, a smart phone, a mobiledevice, a network router, a network switch, a network bridge, or anymachine capable of executing the instructions 816, sequentially orotherwise, that specify actions to be taken by the machine 800. Further,while only a single machine 800 is illustrated, the term “machine” shallalso be taken to include a collection of machines 800 that individuallyor jointly execute the instructions 816 to perform any one or more ofthe methodologies discussed herein.

The machine 800 includes processors 810, memory 830, and input/output(I/O) components 850 configured to communicate with each other such asvia a bus 802. In an example embodiment, the processors 810 (e.g., acentral processing unit (CPU), a reduced instruction set computing(RISC) processor, a complex instruction set computing (CISC) processor,a graphics processing unit (GPU), a digital signal processor (DSP), anapplication-specific integrated circuit (ASIC), a radio-frequencyintegrated circuit (RFIC), another processor, or any suitablecombination thereof) may include, for example, a processor 812 and aprocessor 814 that may execute the instructions 816. The term“processor” is intended to include multi-core processors 810 that maycomprise two or more independent processors (sometimes referred to as“cores”) that may execute instructions 816 contemporaneously. AlthoughFIG. 8 shows multiple processors 810, the machine 800 may include asingle processor with a single core, a single processor with multiplecores (e.g., a multi-core processor), multiple processors with a singlecore, multiple processors with multiple cores, or any combinationthereof.

The memory 830 may include a main memory 832, a static memory 834, and astorage unit 836, all accessible to the processors 810 such as via thebus 802. The main memory 832, the static memory 834, and the storageunit 836 store the instructions 816 embodying any one or more of themethodologies or functions described herein. The instructions 816 mayalso reside, completely or partially, within the main memory 832, withinthe static memory 834, within the storage unit 836, within at least oneof the processors 810 (e.g., within the processor's cache memory), orany suitable combination thereof, during execution thereof by themachine 800.

The I/O components 850 include components to receive input, provideoutput, produce output, transmit information, exchange information,capture measurements, and so on. The specific I/O components 850 thatare included in a particular machine 800 will depend on the type ofmachine. For example, portable machines such as mobile phones willlikely include a touch input device or other such input mechanisms,while a headless server machine will likely not include such a touchinput device. It will be appreciated that the I/O components 850 mayinclude many other components that are not shown in FIG. 8. The I/Ocomponents 850 are grouped according to functionality merely forsimplifying the following discussion and the grouping is in no waylimiting. In various example embodiments, the I/O components 850 mayinclude output components 852 and input components 854. The outputcomponents 852 may include visual components (e.g., a display such as aplasma display panel (PDP), a light emitting diode (LED) display, aliquid crystal display (LCD), a projector, or a cathode ray tube (CRT)),acoustic components (e.g., speakers), other signal generators, and soforth. The input components 854 may include alphanumeric inputcomponents (e.g., a keyboard, a touch screen configured to receivealphanumeric input, a photo-optical keyboard, or other alphanumericinput components), point-based input components (e.g., a mouse, atouchpad, a trackball, a joystick, a motion sensor, or another pointinginstrument), tactile input components (e.g., a physical button, a touchscreen that provides location and/or force of touches or touch gestures,or other tactile input components), audio input components (e.g., amicrophone), and the like.

Communication may be implemented using a wide variety of technologies.The I/O components 850 may include communication components 864 operableto couple the machine 800 to a network 880 or devices 870 via a coupling882 and a coupling 872, respectively. For example, the communicationcomponents 864 may include a network interface component or anothersuitable device to interface with the network 880. In further examples,the communication components 864 may include wired communicationcomponents, wireless communication components, cellular communicationcomponents, and other communication components to provide communicationvia other modalities. The devices 870 may be another machine or any of awide variety of peripheral devices (e.g., a peripheral device coupledvia a universal serial bus (USB)). For example, as noted above, themachine 800 may correspond to any one of the access management system110, the compute service manager 112, the execution platform 114, theauthentication and identity management system 118 and the devices 870may include the computing device 206 or any other computing devicedescribed herein as being in communication with the network-based datawarehouse system 102 or the cloud storage provider system 104.

Executable Instructions and Machine Storage Medium

The various memories (e.g., 830, 832, 834, and/or memory of theprocessor(s) 810 and/or the storage unit 836) may store one or more setsof instructions 816 and data structures (e.g., software) embodying orutilized by any one or more of the methodologies or functions describedherein. These instructions 816, when executed by the processor(s) 810,cause various operations to implement the disclosed embodiments.

As used herein, the terms “machine-storage medium,” “device-storagemedium,” and “computer-storage medium” mean the same thing and may beused interchangeably in this disclosure. The terms refer to a single ormultiple storage devices and/or media (e.g., a centralized ordistributed database, and/or associated caches and servers) that storeexecutable instructions and/or data. The terms shall accordingly betaken to include, but not be limited to, solid-state memories, andoptical and magnetic media, including memory internal or external toprocessors. Specific examples of machine-storage media, computer-storagemedia, and/or device-storage media include non-volatile memory,including by way of example semiconductor memory devices, e.g., erasableprogrammable read-only memory (EPROM), electrically erasableprogrammable read-only memory (EEPROM), field-programmable gate arrays(FPGAs), and flash memory devices; magnetic disks such as internal harddisks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROMdisks. The terms “machine-storage media,” “computer-storage media,” and“device-storage media” specifically exclude carrier waves, modulateddata signals, and other such media, at least some of which are coveredunder the term “signal medium” discussed below.

Transmission Medium

In various example embodiments, one or more portions of the network 980may be an ad hoc network, an intranet, an extranet, a virtual privatenetwork (VPN), a local-area network (LAN), a wireless LAN (WLAN), awide-area network (WAN), a wireless WAN (WWAN), a metropolitan-areanetwork (MAN), the Internet, a portion of the Internet, a portion of thepublic switched telephone network (PSTN), a plain old telephone service(POTS) network, a cellular telephone network, a wireless network, aWi-Fi® network, another type of network, or a combination of two or moresuch networks. For example, the network 880 or a portion of the network880 may include a wireless or cellular network, and the coupling 882 maybe a Code Division Multiple Access (CDMA) connection, a Global Systemfor Mobile communications (GSM) connection, or another type of cellularor wireless coupling. In this example, the coupling 882 may implementany of a variety of types of data transfer technology, such as SingleCarrier Radio Transmission Technology (1×RTT), Evolution-Data Optimized(EVDO) technology, General Packet Radio Service (GPRS) technology,Enhanced Data rates for GSM Evolution (EDGE) technology, thirdGeneration Partnership Project (3GPP) including 3G, fourth generationwireless (4G) networks, Universal Mobile Telecommunications System(UMTS), High-Speed Packet Access (HSPA), Worldwide Interoperability forMicrowave Access (WiMAX), Long Term Evolution (LTE) standard, othersdefined by various standard-setting organizations, other long-rangeprotocols, or other data transfer technology.

The instructions 816 may be transmitted or received over the network 880using a transmission medium via a network interface device (e.g., anetwork interface component included in the communication components864) and utilizing any one of a number of well-known transfer protocols(e.g., hypertext transfer protocol (HTTP)). Similarly, the instructions816 may be transmitted or received using a transmission medium via thecoupling 872 (e.g., a peer-to-peer coupling) to the devices 870. Theterms “transmission medium” and “signal medium” mean the same thing andmay be used interchangeably in this disclosure. The terms “transmissionmedium” and “signal medium” shall be taken to include any intangiblemedium that is capable of storing, encoding, or carrying theinstructions 816 for execution by the machine 800, and include digitalor analog communications signals or other intangible media to facilitatecommunication of such software. Hence, the terms “transmission medium”and “signal medium” shall be taken to include any form of modulated datasignal, carrier wave, and so forth. The term “modulated data signal”means a signal that has one or more of its characteristics set orchanged in such a manner as to encode information in the signal.

Computer-Readable Medium

The terms “machine-readable medium,” “computer-readable medium,” and“device-readable medium” mean the same thing and may be usedinterchangeably in this disclosure. The terms are defined to includeboth machine-storage media and transmission media. Thus, the termsinclude both storage devices/media and carrier waves/modulated datasignals.

The various operations of example methods described herein may beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Similarly, the methods described hereinmay be at least partially processor-implemented. For example, at leastsome of the operations of the methods 300 and 400 may be performed byone or more processors. The performance of certain of the operations maybe distributed among the one or more processors, not only residingwithin a single machine, but also deployed across a number of machines.In some example embodiments, the processor or processors may be locatedin a single location (e.g., within a home environment, an officeenvironment, or a server farm), while in other embodiments theprocessors may be distributed across a number of locations.

Although the embodiments of the present disclosure have been describedwith reference to specific example embodiments, it will be evident thatvarious modifications and changes may be made to these embodimentswithout departing from the broader scope of the inventive subjectmatter. Accordingly, the specification and drawings are to be regardedin an illustrative rather than a restrictive sense. The accompanyingdrawings that form a part hereof show, by way of illustration, and notof limitation, specific embodiments in which the subject matter may bepracticed. The embodiments illustrated are described in sufficientdetail to enable those skilled in the art to practice the teachingsdisclosed herein. Other embodiments may be used and derived therefrom,such that structural and logical substitutions and changes may be madewithout departing from the scope of this disclosure. This DetailedDescription, therefore, is not to be taken in a limiting sense, and thescope of various embodiments is defined only by the appended claims,along with the full range of equivalents to which such claims areentitled.

Such embodiments of the inventive subject matter may be referred toherein, individually and/or collectively, by the term “invention” merelyfor convenience and without intending to voluntarily limit the scope ofthis application to any single invention or inventive concept if morethan one is in fact disclosed. Thus, although specific embodiments havebeen illustrated and described herein, it should be appreciated that anyarrangement calculated to achieve the same purpose may be substitutedfor the specific embodiments shown. This disclosure is intended to coverany and all adaptations or variations of various embodiments.Combinations of the above embodiments, and other embodiments notspecifically described herein, will be apparent, to those of skill inthe art, upon reviewing the above description.

In this document, the terms “a” or “an” are used, as is common in patentdocuments, to include one or more than one, independent of any otherinstances or usages of “at least one” or “one or more.” In thisdocument, the term “or” is used to refer to a nonexclusive or, such that“A or B” includes “A but not B,” “B but not A,” and “A and B,” unlessotherwise indicated. In the appended claims, the terms “including” and“in which” are used as the plain-English equivalents of the respectiveterms “comprising” and “wherein.” Also, in the following claims, theterms “including” and “comprising” are open-ended; that is, a system,device, article, or process that includes elements in addition to thoselisted after such a term in a claim is still deemed to fall within thescope of that claim.

EXAMPLES

Example 1 is a network-based data warehouse system comprising: at leastone hardware processor; and a memory storing instructions that cause theat least one hardware processor to perform operations comprising:creating, in a database, a storage integration object identifying astorage location in a storage platform of a cloud storage providersystem and a cloud identity object, the cloud identity object beingassociated, at the storage platform of the cloud storage provider, witha proxy identity object granted permission to access the storagelocation; creating, in the database, an external stage object based onthe storage integration object, the external stage object identifyingthe storage location and including association with the storageintegration object; receiving, from a computing device, a command toload or unload data at the storage location; and in response to thecommand, loading or unloading, via the proxy identity object, the dataat the storage location using the external stage object.

In Example 2 the subject matter of Example 1 optionally furthercomprises: setting usage permissions associated with the integrationobject; and setting usage permissions associated with the external stageobject.

In Example 3, the subject matter of any one of Examples 1 and 2optionally further comprises: granting a first user permission to usethe storage integration object and granting a second user permission touse the external stage object.

In Example 4, the subject matter of any one of Examples 1-3 optionallyfurther comprises: receiving a command to create the storage integrationobject, the command identifying the storage location and the cloudstorage provider system, wherein the creating of the storage integrationobject is based on the command to create the storage integration object.

In Example 5, the subject matter of any one of Examples 1-4, optionallyfurther comprises receiving a command to create the external stageobject, the command comprising an identifier corresponding to thestorage location and an identifier corresponding to the integrationobject.

Example 6 comprises the subject matter of any one of Examples 1-5,wherein the loading or unloading of the data at the storage locationoptionally comprises: accessing first security credentials for accessingthe proxy identity object using second security credentials associatedwith the cloud identity object; and accessing the proxy identity objectusing security credentials.

In Example 7, the subject matter of any one of Examples 1-6 optionallyfurther comprises: transmitting, to an access management system of thecloud storage provider system, a request for the first securitycredentials, the request comprising the second security credentialsassociated with the cloud identity object.

In Example 8, the subject matter of any one of Examples 1-7 optionallyfurther comprises verifying that the storage location is allowed by thestorage integration object based on information included in the storageintegration object.

In Example 9, the subject matter of any one of Examples 1-8 optionallyfurther comprises verifying user permissions of a user associated withthe command.

In Example 10, the subject matter of any one of Examples 1-9, optionallyfurther comprises verifying that the user has permission to use thestorage integration object; and verifying that the user has permissionto use the external stage object.

Example 11 comprises the subject matter of any one of Examples 1-10,wherein the storage integration object optionally includes a firstidentifier corresponding to the storage location; and the external stageobject optionally includes a second identifier corresponding to aportion of the storage location.

Example 12 is a method comprising: creating, by one or more hardwareprocessors of a machine, a storage integration object in a database of anetwork-based data warehouse system, the storage integration objectidentifying a storage location in a storage platform of the cloudstorage provider system and a cloud identity object, the cloud identityobject being associated, at the storage platform of the cloud storageprovider, with a proxy identity object granted permission to access thestorage location; creating, in the database, an external stage objectbased on the storage integration object, the external stage objectidentifying the storage location and including association with thestorage integration object; receiving, from a computing device, acommand to load or unload data at the storage location; and in responseto the command, loading or unloading, via the proxy identity object, thedata at the storage location using the external stage object.

In Example 13 the subject matter of Example 12 optionally furthercomprises: setting usage permissions associated with the integrationobject; and setting usage permissions associated with the external stageobject.

In Example 14, the subject matter of any one of Examples 12 and 13optionally further comprises: granting a first user permission to usethe storage integration object and granting a second user permission touse the external stage object.

In Example 15, the subject matter of any one of Examples 12-14optionally further comprises: receiving a command to create the storageintegration object, the command identifying the storage location and thecloud storage provider system, wherein the creating of the storageintegration object is based on the command to create the storageintegration object.

In Example 16, the subject matter of any one of Examples 12-15,optionally further comprises receiving a command to create the externalstage object, the command comprising an identifier corresponding to thestorage location and an identifier corresponding to the integrationobject.

Example 17 comprises the subject matter of any one of Examples 12-16,wherein the loading or unloading of the data at the storage locationoptionally comprises: accessing the proxy identity object using thesecurity credentials associated with the proxy identity object.

In Example 18, the subject matter of any one of Examples 12-17optionally further comprises verifying user permissions of a userassociated with the command.

In Example 19, the subject matter of any one of Examples 12-18optionally further comprises verifying that the user has permission touse the storage integration object; or verifying that the user haspermission to use the external stage object.

Example 20 is a computer-storage medium comprising instructions that,when executed by a processing device, configure the processing device toperform operations comprising: creating, by one or more hardwareprocessors of a machine, a storage integration object in a database of anetwork-based data warehouse system, the storage integration objectidentifying a storage location in a storage platform of the cloudstorage provider system and a cloud identity object, the cloud identityobject being associated, at the storage platform of the cloud storageprovider, with a proxy identity object granted permission to access thestorage location; creating, in the database, an external stage objectbased on the storage integration object, the external stage objectidentifying the storage location and including association with thestorage integration object; receiving, from a computing device, acommand to load or unload data at the storage location; and in responseto the command, loading or unloading, via the proxy identity object, thedata at the storage location using the external stage object.

In Example 21, the subject matter of Example 20 optionally furthercomprises: setting usage permissions associated with the integrationobject; and setting usage permissions associated with the external stageobject.

In Example 22, the subject matter of any one or more of Examples 20 and21 optionally further comprises receiving a command to create thestorage integration object, the command identifying the storage locationand the cloud storage provider system, wherein the creating of thestorage integration object is based on the command to create the storageintegration object.

In Example 23, the subject matter of any one or more of Examples 20-22optionally further comprises receiving a command to create the externalstage object, the command comprising an identifier corresponding to thestorage location and an identifier corresponding to the integrationobject.

Example 24 comprises the subject matter of any one of Examples 20-23,wherein the loading or unloading of the data at the storage locationoptionally comprises accessing first security credentials for accessingthe proxy identity object using second security credentials associatedwith the cloud identity object; and accessing the proxy identity objectusing security credentials.

In Example 25, the subject matter of any one of Examples 20-24,optionally further comprises verifying that the storage location isallowed by the storage integration object based on information includedin the storage integration object.

1. A database system comprising: at least one hardware processor; and atleast one memory storing instructions that cause the at least onehardware processor to perform operations comprising: receiving, from acomputing device, a command to load or unload data at a storage locationin a storage platform of a cloud storage provider system; identifying astorage integration object associated with the storage location, thestorage integration object describing properties of a storageintegration between the database system and the storage location in thestorage platform of the cloud storage provider system, the storageintegration object including an identifier of the storage location; andloading or unloading the data at the storage location in the storageplatform based on information included in the storage integrationobject.
 2. The system of claim 1, wherein the operations furthercomprise verifying that the storage location is allowed by the storageintegration object based on information included in the storageintegration object.
 3. The system of claim 1, wherein the operationsfurther comprise verifying that a user associated with the command haspermission to use the storage integration object.
 4. The system of claim1, wherein the operations further comprise: receiving a command tocreate the storage integration object, the command identifying thestorage location and the cloud storage provider system; and creating, ina database, the storage integration object in response to the command.5. The system of claim 1, wherein the operations further comprise:setting usage permissions associated with the storage integrationobject, the setting of the usage permissions associated with the storageintegration object comprising granting a user permission to use thestorage integration object.
 6. The system of claim 1, wherein theidentifying of the storage integration object comprises accessing anexternal stage object, the external stage object identifying the storagelocation and including an association with the storage integrationobject.
 7. The system of claim 1, wherein: the command is a firstcommand; and the operations further comprise: receiving a second commandto create an external stage object, the command comprising an identifiercorresponding to the storage location and an identifier corresponding tothe storage integration object; and in response to the command,creating, in a database, an external stage object based on the storageintegration object.
 8. The system of claim 1, wherein the storageintegration object further comprises an identifier of a cloud identityobject that corresponds to a cloud identity that is associated, at thestorage platform of the cloud storage provider system, with a proxyidentity object corresponding to a proxy identity granted permission toaccess the storage location.
 9. The system of claim 8, wherein theloading or unloading of the data at the storage location comprisescausing the cloud identity to assume the proxy identity using securitycredentials obtained from the cloud storage provider system.
 10. Amethod comprising: receiving, from a computing device, a command to loador unload data at a storage location in a storage platform of a cloudstorage provider system; identifying a storage integration objectassociated with the storage location, the storage integration objectdescribing properties of a storage integration between a database systemand the storage location in the storage platform of the cloud storageprovider system, the storage integration object including an identifierof the storage location; and loading or unloading the data at thestorage location in the storage platform based on information includedin the storage integration object.
 11. The method of claim 10, furthercomprising verifying that the storage location is allowed by the storageintegration object based on information included in the storageintegration object.
 12. The method of claim 10, further comprisingverifying that a user associated with the command has permission to usethe storage integration object.
 13. The method of claim 10, furthercomprising: receiving a command to create the storage integrationobject, the command identifying the storage location and the cloudstorage provider system; and creating, in a database, the storageintegration object in response to the command.
 14. The method of claim10, further comprising: setting usage permissions associated with thestorage integration object, the setting of the usage permissionsassociated with the storage integration object comprising granting auser permission to use the storage integration object.
 15. The method ofclaim 10, wherein the identifying of the storage integration objectcomprises accessing an external stage object, the external stage objectidentifying the storage location and including an association with thestorage integration object.
 16. The method of claim 10, wherein: thecommand is a first command; and the method further comprises: receivinga second command to create an external stage object, the commandcomprising an identifier corresponding to the storage location and anidentifier corresponding to the storage integration object; and inresponse to the command, creating, in a database, an external stageobject based on the storage integration object.
 17. The method of claim10, wherein the storage integration object further comprises anidentifier of a cloud identity object that corresponds to a cloudidentity that is associated, at the storage platform of the cloudstorage provider system, with a proxy identity object corresponding to aproxy identity granted permission to access the storage location. 18.The method of claim 17, wherein the loading or unloading of the data atthe storage location comprises causing the cloud identity to assume theproxy identity using security credentials obtained from the cloudstorage provider system.
 19. A non-transitory computer-storage mediumcomprising instructions that, when executed by one or more processors ofa machine, configure the machine to perform operations comprising:receiving, from a computing device, a command to load or unload data ata storage location in a storage platform of a cloud storage providersystem; identifying a storage integration object associated with thestorage location, the storage integration object describing propertiesof a storage integration between a database system and the storagelocation in the storage platform of the cloud storage provider system,the storage integration object including an identifier of the storagelocation; and loading or unloading the data at the storage location inthe storage platform based on information included in the storageintegration object.
 20. The non-transitory computer-storage medium ofclaim 19, wherein the operations further comprise: verifying that thestorage location is allowed by the storage integration object based oninformation included in the storage integration object; and verifyingthat a user associated with the command has permission to use thestorage integration object.